Hinging hyperplanes for regression, classification, and function approximation

A hinge function y = h (x) consists of two hyperplanes continuously joined together at a hinge. In regression (prediction), classification (pattern recognition), and noiseless function approximation, use of sums of hinge functions gives a powerful and efficient altemative to neural networks with compute times several orders of magnitude less than fitting neural networks with a comparable number of parameters. The core of the methodology is a simple and effective method for finding good hinges. 1). Introduction In an M-dimensional space (xl,... , XM), a hinge function y = h (x) consists of two hyperplanes continuously joined together. Taking xo 1 and using to denote the inner product of two vectors, if the two hyperplanes are given by y= x, y=P=-X, they are joined together on (x: ( P W) x = 01 and we refer to A = (3, or any multiple of A, as the hinge for the function. The explicit form of the hinge function is either max ([+ x,px) or min ((+x, .x). Most of the recently introduced methods for nonlinear regression, classification, and function approximation use expansions into sums of basis functions. The basis functions used are "data selected" from a large parametric class of primitive functions. For instance, CART (Breiman et al. [1984]) uses an expansion into indicator functions of multidimensional rectangles with sides parallel to the coordinate axes. Neural network methods use sigmoid functions of linear functions as primitives. The MARS method (Friedman [1991]) uses products of univariate linear spline functions as its primitive class. In this work, the hinge functions form the primitive class. There are good reasons, as given below, for this approach. Let P be any measure with compact support on E(M) and f(x) any sufficiently smooth function. Then we show in section 3, using methods developed by Jones [1991] and Barron [1991] that there is a constant C(f,P) such that for any K, there are hinge functions hl,... , hK with K C if-_hkII2 C. 1 K